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what's this? 
Hil This is me: 
Tuua Evans 


bloq: jvns.co- $ 
twitter: @b@rk 


and in this zine T want to tell you about 










how Т qot 
better at 
debugging 


These are 5 ways I've changed how L think 
about debugging. 


И Remember the buo is happening 


for a. logical reason. 


Тіс never magic. Really. Even when it makes no sense. 


Иве confident 1. caa fix it 


оо 
on maybe this well Tve fixed 
before: is too , a Vot of hard 
nera T: bugs before 


Talk, to my coworkers 


$9 i 





T hope you learned 
Something new. 


Thanks for reading Ҹ 


Thanks to my, partner Kamal foc 
help reviewing and to the amazin 


Monica Dinculescu (© not waldarf ) 
for the cover art. 


To learn more, see: 


k mu blog : JVNS. CA, 
X mu other zines : JVnS.ca./ 2 ines 
k brendangregg . Com 


But really Yeu just need to experiment. 
Try these tools everywhere. See where they 


help you track down bugs and where they don't. 


Se Strace reals 
helped with 
that problem f 


It takes practice, but L find these tools 


both fun and a useful job skill. T hope 
you will too Y 
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Se ction 1°: I/O and 
M. Su stem colls 4x 


Hello, dear reader V In this z ine , there 


are 3 sections of tools that I love. 


For each tool, 1 || tell you why it's useful 


and give an example. Each one is either 


Some of the most basic questions you 
might have when you log into a. 
mis behaving machine are: 


- is this machine writing to ar reading 
from disk? The network? 


- are the programs reading fi les? Which 


files? 
So, were startin with finding out 
which resources are beng wed and 


what our programs are doing. Let's дої 


Vo kk RAR 
Flame (aps G(€. AN Awesome way to 
visualize CPU performance, popularized by 


Brendan Сгеда 5 Flamea raph -pl tool. 
$7 
h 


2 
= 
– 


$ githeb.com /brendangregg | flame grap 


vA 
Here's what They look. like: 






[Ese 20% [ [teeth 28% 
Panda 20% | alligator 80% 


Main 1007s; 


veux ULIS D з з 
They're constructed fram collections (ооой 
thousands) of stack traces sampled from 


а program : The One abov e. means 807, of 
The stack traces stacted with ~ main 


ANd 107% with v SUR ji aligator 


ц 


eat 
You can construct them from ‘perf ‘recording 5 
(see Brendan Gregg's flame graph github foc how) 
but lots of other unrelated tools can Produce 
them too. T. < them. 
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Strace is "ig favourite program. It prints 
eo System cal| your program used. T4 


А Cool Woy to get an overall picture of whot your 
Program 15 doing, and L 9 USING (Т То answer 


questions like ^ which files ace being apened? : 


ОООО E С 
Е ТАА uus 
сео | ореп(" /home/bork/ . config file") = 3 
Ка "the contents of the file") 
I -.. hundreds of lines... 
Pow "172.217.0.163") 
sendto(5, “hi!!") 


WARNING 


V strace can make your pragram 


run 50х slower. Don't run it on your 
Production database 


L cant do justice to strace here, but I have 
a whole other zine aboutit at 


yvns.ca/zines 


Q рест 9 


perfis not simple or elegant. Tt is a weird 
multitool that does a few different , very 


useful things. First, it's a x \ Sam ling = 
= у Profiler / = 


$ sudo perf record python z 


Try running | 
(press. САС after a few second s) 


You can look at the results with: 





f Sudo pect report | 


Mine says it spent 5% of its time inthe 
Py Dict - Get Item Function. Cool' We learned 
а Ting thing about the CPythen interpreters. 


if You use pect to profile 
0. Python program, it ll 
Show you the C functions 

( Symbols) from the CPython 
interpreter, not the 


Python functions. 


perf can be installed ва 
pretty much any Linux 
machine. The exact 
Features + has will 
depend 0^ Your 
kernel version. 















jw уло | 
ор ода) qoy 9. #0 
490 >рач) 1.519% garamod 
-3992 49 $49| гају WYL- 


203G Áq 
Раја mod S! ouo peur 


1058 y xso мо 
doousuado мо 05150 5229 |. 









r узо 5! 4482 ээч /дезло f: wee де 
Казе з!248ә _ 
о wie apr ОР НИНА 


јәшәу Meu "o 5250 fou, M01$SJ9^ PVA +h h~ v 2e 


+ һо ungn :Saiabay 
= 3390 ој моц = 


jdi»s со s) doousuado 
= S440" + mou = 


“umop nok mo|s „че doous uodo 


"amo|$ XQ) una wo) bo)d Лер? 
JNOM эуош мо 920245 014$ x 


95п j5n $ 4997 4, 
4^8 - 34812 эч pinom noA рор 543. op uo? 320735 / 0 






ә ung. juo NOK " uro боја “о fq pavedo ©шәд 


e NO ane pd yin p 


—_ eee — — ——— 


— — = re 
— = 
= — — — o 


ип noh uəyM 


ћ Ја 99 


ју doous uado 


(39 РУ) 


1499 + 


\° Р) a * 
(burrs, Jaway xnu! 204) 970))4 ^ 


49 Snoyos ( Ayqvqoyd s! •бопбиој „под МЕД 
(HANA WA (005: Д ‘spas s1004 


онуојиој S Шәү$А$Озә оло ou} » 


рч! орол 


. erue моммаш о} pa uom Дод Yapas 
бу v! 201. arods әлоҷ ћорр T 56:43 awos 


(Ф эло] YbBNous 426 коор AUYE I роц 51993 
osvo2mous Of S) DUZ Sup VI мо Kw BRZ Un 


ot posue 
So umou y- |?" 570 pou С обе 
G үәшәлхә 5! 294+ |99} \чо- ANU] 
'o s! vad мо рѕәпЬ pout 29msuo of 


ё pads Quisn Оооо 5! uolos Sty 


12 9N10Q haut эло Loum 
"$9|2^2 go SUDILA Одо Mt YO 
Әм уо 19] V pvads swosood 2no[ 


x пад :% VpS 


SSS COE CO Cas Le 
N Я ~ 
section Ал { networking S 


– 
“lr „А у! 


Tve devoted a lot of space inthis zine to 


networking tools, and I want to explain why. 


A lot of the programs l work with 
Communicate over HTTP. 
response 


request my 
« = =>» name : ово", 
cars 


Every programming language uses the 
same network protocols Y So the network 
IS a nice languag e - inde pendent place to 
answer questions like: 


+ Was the request Wrong , Or WAS 
it the response? 
*(S my service even running ? 


* my program is slow. Whose fault 


is that ? 
Let's д0 ү 


wireshark 
Wireshark is an {amazing GUL tool 


for network analysis, Here's an exercise to 


d eurn LN Run this: 


. —– —_- — –— _— 
— — = — — — — 


| sudo tcpdump port 80 -w http.pcap | 


— — — 
——— ——— – _ = — —  —— у ay a 


While that's ronning, open metafilter. com in 
gour browser. Then press Ctrl+C to stop Тердотр. 
Now we havea pcap file to analyze! 

pes aE шш 


Explore the Wireshark interface | 
Questions You can try te answer > 


O What HTTP headers did uoue browser 
Send to metafilter.com? 


Chint: search | Frame contains "GET |) 
(2) How long did the longest request take? 
(hint: click Statistics > А 


@ How man packets Were exchanged 
with metafilter. com © $есуе($ Poen 
(Nn: search "Тр. бз == 54.186.18.33} \ 
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жх netstat 


Every network request gets sent to 

О. port (like 80) on a computer. To 
receive а. request, a program C a.ka" server") 
needs to be listening onthe fort. Finding 
OUT which programs are istening On which 
ports is really easy . Its just 


xo 
5 coc 
NN n Ne? & 
dK tuna, please V ў RS А 
ny 
also known aS ____ p 
( анай = == = - =  —– — 
( 


~ 


~ — — 


ear netstat -tunapl | 


~ — ~ 
— = = = = 


Here's what you'll see: 
proto ^ loco! address 
tcp 0. pon Е 
So! T V netstat because 

it tells me which processes 
оге running on which ports. 


PID / program name 
2993 / python 


= — — 


наде оне \ 
On 05 X , use | lsof -i -P_i instead, 


— a — 





nar ep 15 д Favour ite starter network 
SPY tool t Try it right now! Run: 


! sudo ngrep d опу metafilter! 
Ix а 


\ 


T = = ere’ – — —- — 
bd Tee - –— - Kee — 


Then go to http: / metafilter. com 
in youe browser. You should see | 
mat ch ing network packets in ngreps 
output Ù We are SPIES MU 


Recently at work T'd made a chang e 
То a client so that it sent 


(*some.id ":,,. 4 with all its 
requests. T wanted to make sure 


iF was working, so T. ran: 


Ci mem ee ee ы eet са n — WS 
; sudo ngrep some- id У 


2 


a 


~ — — = =  — 


1 found cut that everything was ok A 


